NAME¶

libnbd - network block device (NBD) client library in userspace

SYNOPSIS¶

 #include <libnbd.h>
 
 struct nbd_handle *nbd;
 char buf[512];
 
 if ((nbd = nbd_create ()) == NULL ||
     nbd_connect_tcp (nbd, "server.example.com", "nbd") == -1 ||
     nbd_pread (nbd, buf, sizeof buf, 0, 0) == -1)
   fprintf (stderr, "%s\n", nbd_get_error ());
   nbd_close (nbd);
   exit (EXIT_FAILURE);
 }
 nbd_close (nbd);
 cc prog.c -o prog -lnbd
or:
 cc prog.c -o prog `pkg-config libnbd --cflags --libs`

DESCRIPTION¶

Network Block Device (NBD) is a network protocol for accessing block devices over the network. Block devices are hard disks and things that behave like hard disks such as disk images and virtual machines.

Libnbd is a client library for the NBD protocol which can access most of the features of NBD while being simple to use and powerful.

This manual page gives an overview of libnbd, using C as an example, but the library is available from other programming languages.

nbd_create(3), nbd_pread(3), etc.

Each manual page covers one function from the C API in detail. There is a full list in section "C API" below.

libnbd-ocaml(3)

Using the API from OCaml.

libnbd-golang(3)

Using the API from Go.

nbdsh(1)

Using the NBD shell (nbdsh) for command line and Python scripting.

HANDLES¶

To use the API at all you must first open a handle by calling nbd_create(3) (or its equivalent in other languages):

 struct nbd_handle *nbd;
 
 nbd = nbd_create ();

This creates and returns a handle, which is associated with one connection to an NBD server, initially not connected.

Each handle is a complex state machine which can be in states such as created, connected to a remote server, handshaking, idle and ready to issue commands, or busy sending or receiving commands.

There are two levels of API available. A simple high level synchronous API lets you give the handle high level instructions like “connect to the server”, “read a block”, “write a block”, etc. Each of these functions will run to completion, blocking the current thread before returning. A more complicated low level non-blocking asynchronous API is also available where you can integrate with poll(2) or another main loop.

You can freely mix the two APIs on the same handle. You can also call APIs on a single handle from multiple threads. Single API calls on the handle are atomic — they either take a lock on the handle while they run or are careful to access handle fields atomically.

Libnbd does not create its own threads.

USING THE SYNCHRONOUS (“HIGH LEVEL”) API¶

This is the simplest way to use the API, with the possible drawback that each libnbd function blocks until it is finished.

Create a handle and connect to the server:

 struct nbd_handle *nbd;
 
 nbd = nbd_create ();
 if (!nbd) {
   fprintf (stderr, "%s\n", nbd_get_error ());
   nbd_close (nbd);
   exit (EXIT_FAILURE);
 }
 if (nbd_connect_tcp (nbd, "server.example.com", "nbd") == -1) {
   fprintf (stderr, "%s\n", nbd_get_error ());
   nbd_close (nbd);
   exit (EXIT_FAILURE);
 }

Read the first sector (512 bytes) from the NBD export:

 char buf[512];
 
 if (nbd_pread (nbd, buf, sizeof buf, 0, 0) == -1) {
   fprintf (stderr, "%s\n", nbd_get_error ());
   nbd_close (nbd);
   exit (EXIT_FAILURE);
 }

Close the handle:

 nbd_close (nbd);

You can call the high level API from multiple threads, but each libnbd API call takes a lock on the handle and so commands will not run in parallel.

USING THE ASYNCHRONOUS (“LOW LEVEL”) API¶

The low level API is useful if you want to use libnbd in non-blocking code; or if you want to issue commands in parallel from multiple threads; or if you need more control especially over having multiple commands in-flight on a single connection.

To use the low level API you will need to integrate with poll(2) or another “main loop” such as the GLib main event loop.

 int64_t cookie;
 
 cookie = nbd_aio_pread (nbd, buf, sizeof buf,
                         NBD_NULL_COMPLETION, 0);
 if (cookie == -1) {
   fprintf (stderr, "%s\n", nbd_get_error ());
   nbd_close (nbd);
   exit (EXIT_FAILURE);
 }

 int fd = nbd_aio_get_fd (nbd);

 int dir = nbd_aio_get_direction (nbd);

 int r = 0;
 
 dir = nbd_aio_get_direction (nbd);
 
 if ((dir & LIBNBD_AIO_DIRECTION_READ) &&
                 a_read_event_occurred ())
   r = nbd_aio_notify_read (nbd);
 else if ((dir & LIBNBD_AIO_DIRECTION_WRITE) &&
                 a_write_event_occurred ())
   r = nbd_aio_notify_write (nbd);
 
 if (r == -1) {
   fprintf (stderr, "%s\n", nbd_get_error ());
   // ...
 }

 struct nbd_handle *nbd;
 
 nbd = nbd_create ();
 if (nbd == NULL) {
   fprintf (stderr, "%s\n", nbd_get_error ());
   nbd_close (nbd);
   exit (EXIT_FAILURE);
 }

 int fd;
 struct sockaddr_un addr;
 socklen_t len;
 
 /* some code to set up addr,
    then ... */
 if (nbd_aio_connect (nbd, &addr, len) == -1) {
   fprintf (stderr, "%s\n", nbd_get_error ());
   nbd_close (nbd);
   exit (EXIT_FAILURE);
 }
 while (! nbd_aio_is_ready (nbd)) {
   if (nbd_poll (nbd, -1) == -1) {
     fprintf (stderr, "%s\n", nbd_get_error ());
     nbd_close (nbd);
     exit (EXIT_FAILURE);
   }
 }

 int64_t cookie;
 char buf[512];
 
 cookie = nbd_aio_pread (nbd, buf, sizeof buf, offset,
                         NBD_NULL_COMPLETION, 0);
 if (cookie == -1) {
   fprintf (stderr, "%s\n", nbd_get_error ());
   nbd_close (nbd);
   exit (EXIT_FAILURE);
 }
 while (! nbd_aio_command_completed (nbd, cookie)) {
   if (nbd_poll (nbd, -1) == -1) {
     fprintf (stderr, "%s\n", nbd_get_error ());
     nbd_close (nbd);
     exit (EXIT_FAILURE);
   }
 }

ERROR HANDLING¶

When any API call returns an error ("-1" or "NULL" depending on the API), an error message and sometimes an errno value are available. You can retrieve the error message and/or errno of the most recently failed call using nbd_get_error(3) and nbd_get_errno(3). For example:

 if (nbd_connect_tcp (nbd, "remote", "nbd") == -1) {
   fprintf (stderr,
            "failed to connect to remote server: %s (errno = %d)\n",
            nbd_get_error (), nbd_get_errno ());
 }

These functions use thread-local storage to return the most recent error in the current thread. This is why you don't need to pass the handle to these calls. They even work if nbd_create(3) returns "NULL" when there is no handle at all.

For this reason you cannot call them from a different thread. You should call them immediately after the failed API call, from the same thread. Furthermore the error string returned by nbd_get_error(3) is only valid until the next libnbd API call in the current thread, so if you need to keep the string you must copy it (eg. using strdup(3)).

DEBUGGING MESSAGES¶

Libnbd can print lots of debugging messages, useful if you have a problem with the library. Either enable debugging after creating the handle:

 nbd = nbd_create ();
 nbd_set_debug (nbd, true);

or set the "LIBNBD_DEBUG=1" environment variable which will enable debugging by default on all new handles.

Debugging messages are sent to stderr by default, but you can redirect them to a logging system using nbd_set_debug_callback(3).

CONNECTING TO LOCAL OR REMOTE NBD SERVERS¶

There are several ways to connect to NBD servers, and you can even run a server from libnbd. Normally you would connect to a server which is already running, over a local Unix domain socket or a remote TCP connection. The high level API calls are:

 nbd_connect_unix (nbd, "socket");
 nbd_connect_tcp (nbd, "localhost", "nbd");

For nbd_connect_tcp(3) the third parameter is the port name or number, which can either be a name from /etc/services or the port number as a string (eg. "10809").

 nbd_connect_uri (nbd, "nbd://example.com/");
 nbd_connect_uri (nbd, "nbds+unix:///export?socket=/tmp/nbd.sock");

 char *argv[] = { "nbdkit", "-s", "--exit-with-parent",
                            "memory", "1G", NULL };
 nbd_connect_command (nbd, argv);

CONTROLLING NEGOTIATION¶

By default, when beginning a connection, libnbd will handle all negotiation with the server, using only the configuration (eg. nbd_set_export_name(3) or nbd_add_meta_context(3)) that was requested before the connection attempt; this phase continues until nbd_aio_is_connecting(3) no longer returns true, at which point, either data commands are ready to use or else the connection has failed with an error.

But there are scenarios in which it is useful to also control the handshaking commands sent during negotiation, such as asking the server for a list of available exports prior to selecting which one to use. This is done by calling nbd_set_opt_mode(3) before connecting; then after requesting a connection, the state machine will pause at nbd_aio_is_negotiating(3) at any point that the user can decide which handshake command to send next. Note that the negotiation state is only reachable from newstyle servers; older servers cannot negotiate and will progress all the way to the ready state.

When the negotiating state is reached, you can initiate option commands such as nbd_opt_list(3) or their asynchronous equivalents, as well as alter configuration such as export name that previously had to be set before connection. Since the NBD protocol does not allow parallel negotiating commands, no cookie is involved, and you can track completion of each command when the state is no longer nbd_aio_is_connecting(3). If nbd_opt_go(3) fails but the connection is still live, you will be back in negotiation state, where you can request a different export name and try again. Exiting the negotiation state is only possible with a successful nbd_opt_go(3) which moves to the data phase, or nbd_opt_abort(3) which performs a clean shutdown of the connection by skipping the data phase.

EXPORTS AND FLAGS¶

It is possible for NBD servers to serve different content on different “exports”. For this you must pass the right export name to the server. Call this API before connecting:

 nbd_set_export_name (nbd, "export");

Note that there are some servers (like nbdkit(1) ≤ 1.14) which ignore this, and other servers (like qemu-nbd(8)) which require it to be set correctly but cannot serve different content.

These APIs are also available after a successful nbd_opt_info(3) during the negotiation phase, if you used nbd_set_opt_mode(3) prior to connecting.

 int is_read_only = nbd_is_read_only (nbd);
 int can_flush = nbd_can_flush (nbd);

 int64_t size = nbd_get_size (nbd);

DATA COMMANDS¶

You can read and write data from the NBD server using nbd_pread(3) and nbd_pwrite(3) or their asynchronous equivalents.

All data commands support a "flags" argument (mandatory in C, but optional in languages where it can default to 0). For convenience, the constant "LIBNBD_CMD_FLAG_MASK" is defined with the set of flags currently recognized by libnbd, where future NBD protocol extensions may result in additional flags being supported; but in general, specific data commands only accept a subset of known flags.

Libnbd defaults to performing some client-side sanity checking in each of its data commands; for example, attempts to write to a server that has advertised a read-only connection are rejected. It is possible to override aspects of this checking by using nbd_set_strict_mode(3).

Some servers also support:

trim/discard

If nbd_can_trim(3) returns true, nbd_trim(3) can be used to “punch holes” in the backing storage of the disk on the server. Normally (although not in every case) the holes read back as zeroes but take up no space.

zeroing

If nbd_can_zero(3) returns true, nbd_zero(3) can be used to efficiently zero parts of the disk without having to send large amounts of zero bytes over the network (as would be necessary if using nbd_pwrite(3)).

This is slightly different from trimming because the backing storage is still allocated. For some storage types this can make future writes more efficient and/or less likely to fail because of out of space errors.

flushing

Some servers can commit data to permanent storage and tell you that this has happened reliably. There are two export flags associated with this: nbd_can_flush(3) and nbd_can_fua(3).

The nbd_flush(3) call (available if nbd_can_flush(3) returns true) flushes all pending writes to disk and does not complete until that operation has finished. It is similar to using sync(2) on POSIX systems.

A more efficient way to achieve this is to set the flag "LIBNBD_CMD_FLAG_FUA" on write-like calls (like write, trim and zero). This flag means the call will not complete until committed to permanent storage, but it does not involve flushing the entire disk.

prefetching

Some servers can prefetch data, making subsequent reads faster. The nbd_cache(3) call (available if nbd_can_cache(3) returns true) is used to prefetch.

block status

Some servers are able to provide information about the various extents within the image, via the notion of one or more meta contexts. The most common meta context is "base:allocation" (available in libnbd.h as "LIBNBD_CONTEXT_BASE_ALLOCATION"), which can be used to learn which portions of a file are allocated or read as zero. Other contexts may be available; for example, qemu-nbd(8) can expose a meta context "qemu:dirty-bitmap:NAME" for tracking which portions of a file are tracked by a qcow2 dirty bitmap.

In order to utilize block status, the client must call nbd_add_meta_context(3) prior to connecting, for each meta context in which it is interested, then check nbd_can_meta_context(3) after connection to see which contexts the server actually supports. If a context is supported, the client can then use nbd_block_status(3) with a callback function that will receive an array of 32-bit integer pairs describing consecutive extents within a context. In each pair, the first integer is the length of the extent, the second is a bitmask description of that extent (for the "base:allocation" context, the bitmask may include "LIBNBD_STATE_HOLE" for unallocated portions of the file, and/or "LIBNBD_STATE_ZERO" for portions of the file known to read as zero).

There is a full example of requesting meta context and using block status available at https://github.com/libguestfs/libnbd/blob/master/interop/dirty-bitmap.c

PERFORMANCE¶

 struct nbd_handle *nbd[4];
 size_t i;
 bool supports_multi_conn;
 
 nbd[0] = nbd_create ();
 nbd_connect_tcp (nbd[0], "server", "10809");
 supports_multi_conn = nbd_can_multi_conn (nbd[0]) > 0;

 if (supports_multi_conn) {
   for (i = 1; i <= 3; ++i) {
     nbd[i] = nbd_create ();
     nbd_connect_tcp (nbd[i], "server", "10809");
   }
 }

ENCRYPTION AND AUTHENTICATION¶

The NBD protocol and libnbd supports TLS (sometimes incorrectly called “SSL”) for encryption of the data stream and authentication of clients and servers. Libnbd defaults to TLS disabled for maximum interoperability. To enable it on a handle you must call nbd_set_tls(3) before connecting.

To allow TLS, but fall back to unencrypted:

 nbd_set_tls (nbd, LIBNBD_TLS_ALLOW);

Use nbd_get_tls_negotiated(3) to find out if TLS negotiation was successful. Avoid "LIBNBD_TLS_ALLOW" if man-in-the-middle attacks are a concern.

The most secure mode is to require TLS and fail to connect if the server does not support it:

 nbd_set_tls (nbd, LIBNBD_TLS_REQUIRE);

It may also be necessary to verify that the server’s identity is correct. For some servers it may be necessary to verify to the server that the client is permitted to connect. This can be done using either X.509 certificates, or TLS Pre-Shared Keys (PSK). Certificates are more secure. PSK is far more convenient, but you must have an existing secure channel to distribute the keys.

 nbd_set_tls_verify_peer (nbd, false);

 nbd_set_tls_certificates (nbd, "/path/to/directory");

 psktool -u username -p keys.psk

 nbd_set_tls_psk_file (nbd, "keys.psk");

 nbd_set_tls_username (nbd, "username");

CALLBACKS¶

Some libnbd calls take callbacks (eg. nbd_set_debug_callback(3), nbd_aio_pread(3)). Libnbd can call these functions while processing.

In the C API these libnbd calls take a structure which contains the function pointer and an optional opaque "void *user_data" pointer:

 nbd_aio_pread (nbd, buf, sizeof buf, offset,
                (nbd_completion_callback) { .callback = my_fn,
                                            .user_data = my_data },
                0);

For optional callbacks, if you don't want the callback, either set ".callback" to "NULL" or use the equivalent macros (such as "NBD_NULL_COMPLETION") defined in "libnbd.h":

 nbd_aio_pread (nbd, buf, sizeof buf, offset,
                NBD_NULL_COMPLETION, 0);

From other languages the structure and opaque pointer are not needed because you can use closures to achieve the same effect.

 void *my_data = malloc (...);
 
 nbd_aio_pread_structured (nbd, buf, sizeof buf, offset,
                (nbd_chunk_callback) { .callback = my_fn,
                                       .user_data = my_data,
                                       .free = free },
                NBD_NULL_CALLBACK(completion),
                0);

  ...
    (nbd_completion_callback) { .callback = NULL,
                                .user_data = my_data,
                                .free = free },
  ...

COMPILING YOUR PROGRAM¶

On most systems, C programs that use libnbd can be compiled like this:

 cc prog.c -o prog -lnbd

To detect if the libnbd library and header file is installed, the preferred method is to use pkg-config(1) or pkgconf(1):

 pkg-config libnbd --exists || fail libnbd is required

In case the library or header file are not installed in the usual system locations, you can compile your program like this, using pkg-config to detect the proper location of libnbd:

 cc prog.c -o prog `pkg-config libnbd --cflags --libs`

External projects which use autoconf and need to check if libnbd is installed should use the "PKG_CHECK_MODULES" macro in configure.ac like this:

 PKG_CHECK_MODULES([LIBNBD], [libnbd])

This will define "@LIBNBD_CFLAGS@" and "@LIBNBD_LIBS@" which you will need to add to your Makefile.am.

For CMake projects use:

 find_package(PkgConfig REQUIRED)
 pkg_check_modules(LIBNBD REQUIRED libnbd)
 target_link_libraries(prog ${LIBNBD_LIBRARIES})
 target_include_directories(prog PUBLIC ${LIBNBD_INCLUDE_DIRS})
 target_compile_options(prog PUBLIC ${LIBNBD_CFLAGS_OTHER})

To compile an external project against a built copy of the libnbd source tree which hasn't been installed, see the ./run script.

ENVIRONMENT VARIABLES¶

"HOME"

Used in some situations to find TLS certificates. See nbd_set_tls_certificates(3).

"LIBNBD_DEBUG"

If this is set to the exact string 1 when the handle is created then debugging is enabled. See "DEBUGGING MESSAGES" above.

"LOGNAME"

The default TLS username. See nbd_set_tls_username(3).

SEE ALSO¶

AUTHORS¶

Eric Blake

Richard W.M. Jones

COPYRIGHT¶

Copyright (C) 2019-2020 Red Hat Inc.

LICENSE¶

This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA